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A specialized research training institute was 
for 58 college professors who were engaged in Consortium 
evelopment (CORD) projects* An attempt was made to diagnose 
eficiencies in existing educational research and 
t competencies among the participants and to provide them 
ntrated experiences to increase their knowledge and skill 
areas as time permitted. These areas included: (1) 

objectives in behavioral criterion form; (2) test 
on; (3) task analysis and instructional specifications; (4) 
nal systems development; (5) research design; (6) data 
and (7) proposal and report writing. This report gives a 
n of the program, and evaluation and general conclusions. A 
e participants, the schedule for the institute, the 

, and the written evaluation form are included in the 
. (A?) 
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PREFACE 



Michlg.n. .nd reported 

Oregon ?h! ® similar Institute held lest year In Monmouth, 

Oregon. The participants for the Traverse City Institute were like those 
at the .Monmouth Institute, college level profee.or. “.‘^CerreXi^d n I 

0 ““like the 

um! ilwdw J r ; : fot the Traverse City Institute 

In ti e I I "f° WO parts: the Instructional aspects of Che institute were 
In the hands of the Teaching Research Division of the Oregon State System 

I h‘'tr tducatlon and all administrative matters of Institute coordination 

ariv.v"erse CUy“ MJchl' education Center, Michigan State University 

■ lly* -^lohlgan. The management functions needed for effective 

tie institute were thus the responsibility of 

the Institute Coordinator at the Michigan State University Center and 

Included providing for Instructional space and facilities, participant 
activities™" llssure time arrangements, and other such 
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SuBunary 



Tlfiera is a general awareness that many small collage academicians 
lack the knvledge and specialized skills lequlred for the conduct of 
instructional research and development. The research trtlaing Institute 
reported herein attempted to diagnose specific deficiencies in existing 
educational research and development competencies among 58 small college 
professors after they had been provided an opportunity to study and review 
such competencies, and then provided them concentrated experiences to 
increase their knowledge and skills in as many areas as time permitted. 

For each participant not demonstrating such competencies at the termina- 
tion of the institute, an individualized program of study was devised in 
eacli of the areas in which he was deficinct so that he could continue to 
pursue those topics on his own. 

The objectives of the Institute were to develop competencies in the 
following areas: 

(a) Specifying objectives in behavioral criterion form 

(b) Test construction 

(c) Task analysis and instructional specifications 

(d) Instructional systems development 

(e) Research Design 

(f) Data Analysis 

(g) Proposal and report writing 

The research training institute was held at Traverse City, Michigan 
with instructional responsibilities given to The Teaching Research Division, 
Oregon State System of Higher Education, and institute management and 
coordination the responsiblity of the Continuing Education Center, Michigan 
State University. 

Participants were provided self-instructional materials for each topic 
and time scheduled for systematic study during the Institute. Each topic was 
then preseoLed in a large group-paced programed sequence. At the completion 
of the group-paced presentation participants were grouped into small seminars 
or individual instructional sessions for additional study. Those individuals 
achieving competence on each topic before the allotted time were encouraged 
and given guidance in preparing either a research proposal or the develop- 
ment of a system of instruction relevant to his area of Interst. 
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Criterion test results as well as written and oral evaluatioaa iadicatad 
that the institute objectives were achieved. 

/' 

It was recommended that if a similar institCrte viere conducted that 

1. The institute should be separated into two sections: 
research and development. 

2. That trainees be screened for interest in either 
research and development and scheduled accordingly. 

3 . That the physical location of the institute be at 
the institution responsible for the instruction. 
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Introduction 



To facilitate the basic objectives of the various consortium research 
development (CORD) projects, a specialized educational research training 
institute was conducted for college professors who were currently partici- 
pating in a research development grant project. This group numbering 58 
professors met at the Continuing Education Center of the Michigan State 
university at Traverse City during the two week period beginning August 18 
and ending on August 30, 1968. The Institute was designed to provide a 
set of learning experiences to the participants which would increase their 
competencies in research methods in order that they might improve their 
courses of instruction through research and enable them to engender 
educational research activities within their spheres of influence, 

Empirical development of courses of instruction requires several 
specialized skills and utilizes a newly evolving knowledge base. The 
diversity of CORD institutions and backgrounds of participating professors 
suggested tliat considerable flexibility be programed into the Institute 
activities. In addition, there was general awareness that many academicians 
lacked knowledge and specialized skills required for the conduct of re- 
search and development activities. Witli this in mind, provisions were 
made to identify specifl'^ deficiencies in existing educational research and 
development competenci' among participants after they had been provided an 
opportunity to review such competencies, and then provided them concentrated 
experiences to increase their knowledge and skill in as many of those de- 
ficient areas as time permitted. If such competencies were not demonstrated 
at the termination of the formal institute, individualized programo of 
study were devised for each participant In each of his need areas, thus 
enabling him to continue to pursue those topics on his own if he desired. 



Objec t ives 

The specific objectives of the institute were to develop competencies 
in the following areas: 



(a) specifying objectives in behavioral criterion form. This included 
the statement of objectives for instructional systems design 
purposes, their modification and refinement in developmental 
phases, and their revision and simplification for dissemination. 

(b) Test construction. This included information on the procedures 
used in the selection and development of appropriate criterion 
measures for various types of behavioral objectives, and tech- 
niques for assessing absolute criterion performance va. relative 
performance. 
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(c) 



id) 



Ta»k analysis and Instructional specifications. This included 
infornatlon on hierarchical Cask analyaia and technique, for 
specifying instructional requirements of various kinds to 
achieve task related behaviors. 

Inatructlonal systems development. This included exparlmantal 
operations and procedures, development of external control 
strategies, instructional monitoring procedures, and evaluation- 
revision operation revisions to maximiac Che learning effects of 
each instructional element. 



(e) Research design. This included information on "within group" 
variance and the identification of learner variables celevant 
to the learning experience, alternative experimental designs 

the manipulable and non-manipulable variables, types of research 
studies, and operations research methods. 

(f) Data analysis. This Included selection of appropriate statisti- 
cal techniques for analysis of evaluation and/or experimental 
data, and the use of computers in the process of analysis. 

(g) Proposal and report writing. This included information as to the 
components of the research proposal, the use of operational 
language, and the key role of evaluation of the educational pro- 
duct, and/or the experiment itself— including limitations, in- 
ferences, Implications, and description of new questions raised 
and additional experimentation required. 



Trainees 



The training group consisted of a total of 62 participants, 58 of whom 
were college professors selected by the various U.S. Office of Education 
Regional Research Program Directors from those institutions of higher learn- 
Ing cooperating in the CORD program, and the remaining four participants, 
U.b. Office of Education observers representing the bureau of Research. 

A complete roster of the institute participants is contained in Appendix A. 

The participants represented a very diverse population of professional 
backgrounds. The breakdown was as follows: 



OE/DHEW 



English 



Higher Education Administration 



Psychology 



History i 



Music 2 



1 

9 

3 
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Education 



Natural Scianca 



Biological Scianca 
Sociology 

Guidanca and Counaaling 
Cheniatry 

Inatitutional Raacarch 



Daan of Studenta 



Child Study 
Diatician 



Other 



9 

2 

2 

3 

2 

2 

9 

2 

1 

1 

9 



DESCRIPTION OF THE PROGRAM 



r^s-institue preparations consisted of two major activities* 

lid*'’ 0^ ln*i.ructlon«X macerlala for u« In tho’lnotltute- 

comunlctlon. to participant, prior to th. ’ 



The In.tructlonal ataff utilized the group and aeif-lnatructlonal 
Mteriala prevloualy developed under a Small Porject Reaearch contract 

de*i"ip“i:„r"‘F:uror;h*: «<*uc.tlona; r.J;rch 

of new ln:?::"uonar.y: eii! ?hi“hr”e 

ri*rfiv 4 ine cnree remaining topics relate most di- 

rectly to planning and conouctlng research on Instructional ayatama and 
reporting research reaulta. The complete set of self-study ' 

“ilahle fioi 0«8°" iMtltute and ate 



umild**ri*****j the nature of the rogram it was expected that there 

In the abilities and experiences of the p.r- 
iid'w V Therefore, the institute was designed to identify strengths 

mufh 'J'" P"“n^P*nt in each topic area and to provid!, aa 

oh as possible, the necessary background experience and the Indlvl- 

is“:h““it;c”?:n“."‘'“^"‘‘ .*P.ri.nc.^or 
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aZII * * cUterlon test, snd saall group or Individual rcmadial 

In .11 topic, befor. thu termination of th. In.tltut. undertook the 
preperetlon of . propo.el for re.e.rch, or the develop««“r.n to- 

zizT" k“r:^o“dirnord«.- 

Behavioral topics and staff members responsible for them were: 



Topic Specialists 



Dr. Casper F. Paulson 
Dr. Paul Twelker 
Dale UamreuB 
Del Schalock 
Dr. Jack Crawford 
Dr. James Beaird 
Jack Crawford 



Dr. 

Dr. 



Dr. 



Behavioral Topics 
Specifying Objectives 

Task Analysis and Instructional Specification 
Instructional Systems Development 
Test Construction (Measurement) 

Research Design 
Data Analysis I & II 
Proposal and Report Writing 

Topics war. covered In s.quenc. .ft.r .n orientation and overvlm 
session. The raphasls of the first week of the in.tltut., a. Indicated 
iLtr* topics, was primarily on the development of am 

devoted primarily to pUnnlng 

Thf on instruction and curriculum innovation., 

nstltute was concluded with a summary and evaluation session. 

I-, '*•* institute progressed the participants began to identify their 

Interests around either the area of development or research. Durlu the 

th.raft!i: '*>••• interests bec«se sufficiently cJSrJfl.d 

eiternoon small group study sessions were formed accordingly. 

on eiil“tior'"“ "• ““<*« *he section 

A typical daily schedule was as follows: 

8:00 to 9;A5 A.M. Group-paced Topic Presentation: This was a carefully 

prepared, pre-tested, learning experience on one of the institute Topics 
requiring active response from participants and providing feedback to 

J percentage of students responding correctly and an 

individual record of errors made by each student. 

9:45 - 10:00 A.M. Break 
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gain comptcencc in th« tpecifUd bthavlor. 




12:00 - 1:30 P.M. Lunch« 

Independent end Smell Group Study: This wee e contlnue- 

tion of the work underteken in the morning. Some Indlvlduele denonetreted 
sufficient competence In the topic to epply the behevlor to the develop- 
ment of a research proposal relevant to his own area of interest. 



7:00 - 10:00 P.M. Independent Study: During this period the participant 

studied the specially prepared self-instructional materials on the topic 
scheduled for the following day. ^ 

The complete institute schedule is contained in Appendix B. 



Three forms of evaluation were employed: 1) a simple post-test measure 

of criterion scores designed to determine what learning gains resulted from 
the institute; 2) participants' written appraisals of the institute on a 
prepared evaluation form; and 3) participants' summary and evaluative commen 
obtained during the final two hour Institute session devoted specifically 
to that purpose. Each of these evaluations are summarized below. 



Post-Test gain sc ores 

During the institute held in Monouth, Oregon the previous year, a 
pre-post-test design was followed for evaluation. This design was dropped 
during the Traverse City Institute in favor of a post-test only design for 
two reasons: (1) base line data had been obtained from the previous 

Institute which could be used to fairly represent the level of ability of 
the participants in the Traverse City Institute in that they ware drawn 
from the same general population; and (2) the time lost and the contamina- 
tion resulting from a pre-test during the two week training period strongly 
urged that the pre-test be dropped. 




4:00 - 5:00 P.M. Group Meeting (Interaction): 



housekeeping details, etc. 



Evaluation 



Th« raiulc* of Monmouth Inotltuto pra-and Travaraa City poat-taat acoraa 
on crltarlon maaauraa for aach Inatltuta tpplc and total acoraa of all 

“*App“d”%!''*” "" ** “• contalnad 



Tab la 1 



Maana Standard Davlationa, Rangaa, and Total Poaaibla Polnta of All 

p^r ! H •“«» ToMl Taat Seoraa for All Monmouth Inttltuta 

Pro- and Travaraa City Poat-taat Maaauraa. 



Criterion 

Mcaeure n 



Standard 

o Daviation Ranae 



Total 

Points 

Possible 



Behavioral Pre- 24 

Objectives Post 35 



31.29 11.16 7-47 

^3»23 4.98 34~-48 



60 

49 



Objective Pre- 24 10.67 1.86 7-14 

^alysis P^t 35 62.57 9.95 45-80 



Instructional Pre- 24 15.08 4.53 

Development Post 35 22.37 .4.98 

Pre- 24 12.52 2.65 

Measurement Post 35 47.65 12 . 08 



10-22 44 

13-32 47 

8-16 44 

16-63 119 



Experimental Pre- 22 1.45 1,77 

Poet 35 5.83 1 ! 26 



Data Pre- 24 6.50 2.52 

Analysis I Post 3 5 11.94 3.33 



1-11 20 

3-18 20 



Data Pre- 24 

Analysis II Post 35 



1*58 2.94 0-8 16 

3.84 3.14 0-12 20 



Total of all Pre- 22 79.38 19.05 41-115 209 

Poet 35 199.44 26.21 130-250 352 



Inspection of Table 1 reveals that post-test scores vbta higher than 
pre-test means in all cases. Although total possible points differed between 
pre- and post-test measures in several instances the trend for all measures 
as inuicated by means and range scores was sufficiently sreat to indicate that 
changes in behavior in the direction desired did occur. 
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Individual Written Evaluations 

The second form of evaluation required participants, at the close 6f 
the Institute, to complete an evaluation form containing seven brjsd questions* 
Participants were requested to (1) Identify which Instructional section 
was most relevant to their professional needs, (2) list ways In which their 
answers to the flrsl question would Influence their sub^iequent Job performance, 
(3) suggest Improvements to the instruction in the Institute, (4) re- 
commend elements to eliminate or de-cmphaslze In the instruction, (5) write 
at least one positive and one negative comment regarding various functions 
of the Institute, (6) react to how, if at all, they might change a para- 
graph they had written at the beginning of the Institute, and (7) comment 
on anything In general about the Institute. 

During the fourth sessloii of the Institute on measurement, participants 
expressed a concern that small group study sessions were not providing 
maximum learning opportunity because of the diverse Interests In the parti- 
cipants. Subsequent study sessions were therefore grouped according to 
participants' major interests of either research or development. The re- 
sults of participants' ratings to the written evaluation have been 
summarized to reflect these groupings and are listed under three headings: 
total group, research Interest group and development Interest group. 

Summaries are shown In Table 2. 



Table 2 

Tallies of Participants' Written S^iimnarles 



TOTAL - 35 
RESEARCH - 21 
DEVELOPMENT - 14 



1. What Instruction did you find most 
relevant to your professional needs? 

Advanced Analysis 

Proposal Development 

Research Design 

Behavioral Objectives 

Measurement 

Teacher-Models 

All sections 

Prototype Development 

Systems Approach 

Evaluation 

Games 

Report Writing 
Personnel Conferences 



Total Research Development 



14 

12 

12 

10 

9 

7 

5 

4 

3 

2 

2 

1 

1 



14 

7 

12 

5 

9 

3 

2 

2 

1 

1 

1 

1 

1 



5 

5 

4 

3 

2 

2 

1 

1 
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Total Rcfftarch 



In what ways will th« things you 
described in (1) influence the 
performance of your job in the 
coming year? 

Motivated to do research in instructional 
methods and counseling techniques. 18 

Improve instructional procedure 8 

Increase awareness of objectives approach 7 

What could be added or changed to make 
the Instruction more useful and effective? 

Small group interaction in learning and 



study 9 
More effective orientation 9 
More visual aids 7 
Better organisation 8 
Rewriting 2 



What do you feel should be eliminated 
or de-emphasized? 



Less time on different systems 12 

Depends upon participants 5 

Repeat of manual 3 

Data analysis optional 2 

Criterion tests for two weeks 2 

Exam in manner of which it was given; 
better to have given sections of it with 
instructional units. 1 



Please write at least one positive and 
one negative comment on each of the 
following: 

(a) reading materials 

Adequate-challenging 18 

Some areas not sufficiently clear H 

Unorganized 10 

(b) small group activities 



Discussion group good 20 
Arrangement poor 6 
Lengthy 3 
Several good options 1 



12 

7 

4 



7 

7 

3 

4 
1 



8 

3 

2 

2 

1 



0 



10 

9 

7 



10 

5 

4 

1 



Davelopmant 



6 

1 

3 



2 

2 

4 

2 

1 



4 

2 

1 

1 



1 



8 

2 

3 



10 

1 

1 

0 



(c) flexibility of the program 

Excelllcnt 

Good 

Rather flexible and subject to justifiable 
change>some staff still had enough ego 
Involvement to show slight hostllites 
at strong ctiticlsra. 

Good during latter part, plus Inventory 
first day could have provided data for 
structuring small groups. 

Quite flexible-allowing participants 
to pursue worthwhile areas of personal 
use and effort to meet needs of everyone 
■'00 rigid first 3 or A days. 

Unending flexibility is not necessary for 
this program-some is needed for meeting 
individual needs. 

Large groups rather inflexible 
Night be bad if too flexible and only 
most outspoken members are needed. 

Too many options at times. 

(d) Interaction with staff and other 
participants . 

Excellent 

Good 



Friendly and sensitive 
Opportunities for informal contacts 
Staff available at all times. 

Poor beginning. 

More Interaction could have been 
encouraged by numerous methods. 
Somewhat distant 

One staff member made you feel inferior 
Few staff carried the burden for all. 

On the first day of the institute, you 
wrote a paragraph on some point of dis- 
agreement with the general orientation 
chapter. How if at all, would you 
modify now what you wrote then? 

No change 

Educational research-more preparation 
Teacher-students, more participation 



Total 

More confidence in teaching direction l 

Detailed information needed before 

workaho?. ^ 

7. There muat be something we've overlooked in 
our evaluation. Please answer (a) or (b) 



(a) 


You must be kidding I 
cramp already. 


I have 


writer's 


7 


(b) 


What I raally wanted 


to tell 


you. 






but you never asked. 


is : 





Teaching Research is a scholarly, creative 
group which has a great deal to offer 
higher education. X 

This has been a wonderful experience for me. 6 
See memo of August 30. X 

Let's have more of these for the people 

who didn't get here. X 

Pre-institute communication about 

program content. X 

Motivation still the most powerful force 
in teaching-not method or strategy which 
ignores it. X 

Two weeks were very enriching. 1 

Institute should have been held where 

there were more library facilities. 1 

Lack of preparation. 1 

Original program should have been 

carried out. X 

Second week right up my alley. Belief 
that I am a better professional as 
a result of attending this workshop. 1 

Cjieat group. X 



Research Development 
1 0 

1 0 

6 1 



1 

3 

0 



0 

3 

1 



1 0 



1 0 



0 1 
0 1 



1 0 
0 1 



0 1 



1 0 
1 0 



From Table 2 several things can be noted. Responses to question one 
indicated that the research interest group found analysis, design, and 
measurement instruction most relevant whereas the development interest group 
found proposal writing and behavioral objectives most relevant. Under 
question five, the research interest group considered the first 3 or 4 
days of the institute (devoted to development issues) as too rigid, whereas 
the development Interest group was apparently satisfied with the flexibility 
provided. 

Both groups indicated receiving strong influence from Che instruction 
in the institute to conduct research in' instruction and counseling when 
they returned to their institutions. 
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Responses to question four indicated that if a single problea or 

through .11 ..ctlon. of th. b.du. 1 
it was felt it would have strengthened the instruction considerably. 



In general, participants rated the reading materials as adequate to 
challenging, the small group activities as good, the flexibility of the 

program as quite flexible, and the staff interaction as good and available 
at all time. 



Group Oral E valuat ion 



The third level of evaluation consisted of a concluding two hour group 
meeting with participants and staff to summarize and evaluate the institute. 
More than fifty different suggestions were made including reference to 
written materials, small group study, scheduling, group interaction, 
institute organization and "warm blanket" services. 

The most notable outcome from this evaluation was the wide differences 
of opinion among the participants in terms of what they liked and disliked 
about the institute. Little agreement was obtained and other than letting 
institute members get things "off their chest", little was gained beyond 
the written evaluations. 



Agreement was expressed, however, that much more value than simple 
research skill attainment emerged from the institute. Participants in- 
dicated that they had acquired a broad understanding of how to better help 
others at their home institutions and expressed a general confidence and 
optimism about plans for writing proposals and carrying out instr^'ctional 
research of their own. 



General Co n clusions 

In general, it was concluded that the institute did achieve its stated 
objectives and even more, as was noted above. 

The following changes are recommended if a similar institute were to be 
conducted . 

1. The institute should be separated into two distinct sections: one 

with a research emphasis and one with a development emphasis. 
Measurement and proposal writing should be common to both. 

2. Trainees should be screened for interest in either research or 
development and scheduled accordingly. Responsibility for select- 
ing trainees should be given to the institution responsible for 
conducting the struction. 



3. The physical location of the institution should be at the Institution 
responsible for conducting the instruction. Ir was difficult in the Traverse 
City Institute for the instructional staff coming from Oregon to anticipate 
all the necessary resource needs before departing for Michigan with the 
result that several desirable items were not available when their need 
became evident. In another sense, many needs cannot be anticipated prior 
to involvement with participants; but if the institute were held at the 
instructional staffs' Institution, all related research and development 
materials in the agency can easily be drawn upon at very short notice. 
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APPENDIX A 



NAME 

Gien E. Alien 
Ronald Appleman 
Franc ! 8 X., Brady 
Charles Bailey 
C. H. C-iapr::an 

William H. Clements 
Charles R. Colvin 
Sister Maureen Connors 

Charles W. Cc.ck 
Mariar;rie Cock 
Stanle,. Davis 
Anti’K Debcrs 
Anthonj M. Deiulio 

Raymond m. luemin^ 

Nancy Dixon , 

John M. Dodd 

Andrew S. Edvards 
Doris Epstein 



LIST CP PARTICIPANTS 
RESEARCH TRAINING INSTITOTE 
Traverse City, AuKust 19 to 30 , I 900 
as C'f Auyust Ik 

ADDRESS 

S. Colorado State College, Pueblo, Colorado ^*1005 
Northern Oklahoma College, Tonkawa, Oklahoma 7^*653 
Elmira Cc liege, Elmira, Ni Y. 

LeMo^ re College, Memphis, Tennessee 

Coldwater Exempted Village Schools, 310 N. Second Street 
Coldwater, Ohio k^E2d 

Wisccnsin State University, Stevens Point, Wis., ^kkbl 

State University College, Predonia, N. Y. 

Ursuline College for Women, 2600 Lander Rd., 

Cleveland, Ohio 44l2i^ 

Carroll College, Waukesha, Wisconsin 

Merrill -Palmer Institute, 71 E. Perry, Detroit, Mich. kSPO 
Ithaca College, Ithaca, New York 
Universit,. c.l Dayttn, Dayton, Ohio k^kO^ 

State University College, Predonia, N. Y. 
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Syn<il«y 
August 18 



Monday 
August 19 



Tuesday 
August 20 



t 21 



Thursday 
August 22 



Friday 
August 23 




APPE3IDIX B 



NATIONAL RESEARCH TRAINING INSTITUTE 



Traverse Cltyp Michigan 
Official Institute Scjjs^iU;^ ^ 




Dormitory check-in o 



8g00°Ug3Q 



1;00-4830 p.mo ' 



£s 



7800“10800 Pom.. 



Section I-Orientation 
(Dto Edling) 



Orientation coot* & Social Event 
Independent Study- 
Specify^g Objectives 



Section I-Practice 
Session-Speci fying 
Objectives 0 
(DTo Paulson) 



Small Grpo Session- Independent Study- 

Specify' g Objectives Objective Analysis 

(Dro Edling-Staff) 



Section I-Practice 
Sess.-Objo Analysis 
(Dro Twelker) 

Section I-Practice 
Sesso-Instro SystoDey. 
(DTo Hamreu/f) 



Small Grpo Session- 
Objective Analysis 



Small Grp. Session- 
Ins true# Systc Devo 
(Dr. IWelker-Stfo) 



Independent Study- 
Ins true. Systo Dev 



Independent Study- 
Measurement 



Section I-Practice 
Sees o -Measure men t 
(Dr, Schalock) 



3:00 Large Grpo Sess.- 
Evaluation (Dr. Paulson) 

Small Grp. Session- indepandent Study- 

Measuremsnt Research Design 

(Dr, Hamreus-Staff) 



*At 4:30 p.m. to 3:00 p.m. daily there vrill be a genaral discussion session 
in the auditorium to evaluate and clarify issues raised during the day. 



Saturday Independent Study - Research Design or Propoeal Preparation 

August 24 



Sunday 
Auguat 25 



Free 








Official Ins tl tute Scheduls 
(Second Week) 





8:00-11:30 a.m. 


1:00-4:30 o.m.* 


7:00-10:00 p.m. 


Monday 
August 26 


Section I-Practlce 
Sess.-Res. Design 
(Dr. Crawford) 


Small Grp. Session- 
Research Design 
(Dr. Schalock-Stf.) 


Independent Study- 
Data Analysis I 


Tuesday 
August 27 


Section 1-Practlce 
Sess.-Data Anal. I 
(Dr. Bealrd) 


Small Grp. Session- 
Data Analysis I 
(Dr. Crawford-Stf .) 


Independent Study- 
Data Analysis II 


Wednesday 
August 28 


Section I-Practlce 
Sess.-Data Anal. II 
(Dr. Bealrd) 


Small Grp. Session- 
Data Analysis II 
(Dr. Crawford-Stf.) 


Ind. Study-Proposal 
& Report Writing 


Thursday 
August 29 


Section I-Practlce 
Sess. -Proposal & 
Report Writing 
(Dr. Crawford) 


Sml. Grp .Sees. -Prop Review all sections 

& Report Writing 
(Dr. Bealrd-Stf.) 

3:00 Research Management 
(Dr. Edllng 




Section II-Sml.Grp. 
Sess.-Data Anl. II 
(Dr. Bealrd-Staff) 


Prac. Sess. -Proposal 
& Report Writing 
(Dr. Crawford) 


Social Event 


Friday 
August 30 


Section I-Flnal 
Criterion Meas. 
(Dr. Edllng) 


Summary & Evaluation 
of Institute- Dormitory 
Check-out 
(Dr. Edllng) 





*At 4:30 poiD. to 5:00 p.m. dally there will be a general discussion session 
In the auditorium to evaluate and clarify Issues raised during the dayo 



Participants may make crave 1 departure schedules 
after 3:00 p.m. Friday. 



National Research Training InatituCe 



STAFF ASSIGNMENTS 
Monday, August 19 

AM - Orientation (Jack) all ataff at 9i30 AM 
PM - Input (Study Behavioral Objectives) 

Tuesday, August 20 

AM - Behavioral Objectives (Bud) 

PM - Snail Group (Jack E. , Jack C. , Jin, Del, Dale) 

Wednesday, August 21 

AM • Objective Analysis (Paul) 

PM - Small Group (Edling, Crawford, B aird, Schalock 4 Paulson) 

Thursday, August 22 

AM - Prototype Development (Dale) 

PM - Small Group (Jack E. , Bud, Paul, Jim, Dale) 

Saturday and Sunday 
Free Time 

Monday, August 26 

AM - Design (Jack C.) 

PM - Small Group (Ed, Bud, Del, Dale) 

Tuesday, August 27 

AM - Data Analysis (Jim) 

PM - Small Group (Ed, Crawford, Del, Bud, Paul) 

Wednesday, August 28 

AM - Data Analysis (Jim) 

PM - Small group (Ed, Del, Bud, Paul, Dale) 

Thursday, August 29 

AM - Proposal (Crawford) 

Management (Jack) 3:00 PM 
PM - Small Group (Ed, Del, Bud, Paul, Jim) 

(Jack C* , Bud, Del, Dale free to leave at 3:00 PM) 

Friday, August 30 

AM - Evaluation (Examination) Jack, Jim, Paul 
PM - Evaluation - Jack, Jin, Paul 
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Behavioral Objectives Test 



1. One way In which a written behavioral objective for teaching may differ 
from a non-behavloral objective Is that the behavioral objective always 
specifies : 

A. Teaching methpds 

B. Teacher behavior 

C. Length of a teaching unit . 

D. Criteria for measurement. 

2. An objective,*’ as it has been defined for the purposes of writing 
behavioral objectives, denotes: 

A. A goal that a teacher intends students to accomplish 

B. A desired goal for students to accomplish 

C. A goal for teachers to accomplish In their teaching methods 

D. A goal teachers would like to accomplish with their students 

3. Generally , the most valid Indications of student behavior that are 
related to a behavioral objective are those which: 

A. Reflect the objective indirectly 

B. Foster democratic ideals 

C. Allow the student to express himself 

D. Are linked directly with the objective 

4. The following verbs might be used In writing behavioral objectives 
concerning the testing of geography. Which verb would require the least 
clarification in a behavioral objective? 

A. Understand 

B . Draw 

C. Locate 

D. Identify 

5. The following verba might be used In writing a behavioral objective 
for teaching high school English. Which verb would require the least 
clarification of a behavioral objective? 

A. Write 

B. Appreciate 

C. Illustrate 

D. Svimmarlze 

6. In a behavioral objective, the audience is: 

A. All the students In a particular grade or level 

B. Some of the students in a particular grade or level 

C. A group of students who are expected to reach the criterion in 
the behavioral objective 

D. A group of students who behaved as the objective Indicates 

7. The "conditions" of a behavioral objective specify 

A. The setting in which the students* behavior Is to occur 

B. The actions which the teacher will observe 

C. The actions of the leader 

D. Criteria for measuring the student behavior 

8. Th. «beU=vior.l" a^«ct of a behavioral obiectlve spaclf less 

A. Teacher behavior 

B . Pupil behavior 

C. Behavioral conditions 

D. Measurement of behavior 




II. 



From each of the following groups of objectives select the one objective 
which Is most nearly stated in behavioral terms. 

N. 

9. A. To teach the students how to build a 3 x 5 Inch jewel box. . . 

B. The student will learn the principles of constructing small boxes... 

C. Each 10th grade shop student will build a 3 x 5 Inch jewel box... 

D. To show 10th grade students the proper way to construct a 
3x5 Inch box. . . 

10. A. To remember the names of the ten provinces of Canada In such a 

way as to... 

B. To learn and remember the names of the ten provinces of Canada... 

C. To name and label the ten provinces of Canada on a blank map showing 

only... 

D. To appreciate the Importance of the ten provinces of Canada... 

11. A. To learn the names of the different latitudes of... 

B. To write on an outline map the names of the different latitudes of... 

C. To know the names of the different latitudes of... 

D. To remember how to identify the different latitudes of... 

12. A. To teach the fundamentals of diagraming electrical circuits... 

B . To learn the fundamentals of diagraming electrical circuits . . . 

C. To diagram an electric circuit with all the fundamentals... 

D. To know how to diagram an electrical circuit. 

13. A. To define the terms decagon» geometry » and equilateral... 

B. To learn the terns decagon, geometry and equilateral... 

C. To know the concepts decagon, geometry and equilateral... 

D. To understand the terms decagon, geometry and equilateral... 

14. A. To explore the Identification of various types of vegetation... 

B. To name and describe In writing ten types of vegeatlon. . . 

C. To learn the names of ten different types of vegetation... 

D. To know the names of ten different types of vegetation... 

15. A. To point out five essr^ntial points on a map... 

B. To learn about five essential points on a map... 

C. To know and understand five essential points on a map... 

D. To appreciate the value of knowing five essential points on a map... 
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in. 

From each of the following groups of behavioral objectives select the one 
that most accurately describes the desired behaviors . 

Locate ten major oceans, bays and straits on an outline map. 
Identify ten major oceans, bays and straits on an outline map. 
Write the names of ten major oceans, bays and straits on an 
outline map. 

Be able to recognize ten major oceans, bays and straits on an 
outline map. 

17. A. To send a four-word message by Morse code with a blink light. 

B. To send a Morse code message. 

C. To send a message with a blink light. 

D. To send a message using a code. 

18. A. Must be able to read Spanish writing. 

B. Must translate Spanish Into English verbally. 

C. Must read a Spanish paragraph and translate orally Into English. 

D. Must be able to tell the differences between languages. 

19. A. Must write a Campbell style library paper of at least ten pages. 

B. Must show an ability to write a library paper. 

C. Must write a Campbell style library paper and finish It. 

D. Must be able to write a paper of ten pages or more. 

20. A. Must give 4 examples of methods used to teach biology. 

B. Must write examples of 4 basic Instructional techniques in biology 

C. Must demonstrate an ability to teach biology 4 different ways. 

D. Must show 4 examples of how to teach biology. 

21. A. Find the ten largest cities in Canada. 

B. Locate the position of each of the ten largest cities In Canada. 

C. Write a list containing the ten largest cities In Canada In order 
of size. 

D. Recognize the rank of each of the ten largest cities in Canada. 

22. A. Write on an Isothermal map with a red pencil accurately, 

B. Find the three spots on an isothermal map with heaviest rainfall. 

C. Mark with a red pencil the 3 areas on an Isothermal map with 
heaviest rainfall. 

D. Locate and recognize areas of heavy rainfall on an Isothermal map. 



16. A. 

B. 

C. 

D. 
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IV. 



From each of the following groups of statements select the one which most 
clearly specifies an acceptable level of performance . 

To write a topic sentence suitable for three given related sentences. 
To write a good topic sentence without error* 

To write accurately a topic sentence in 3 minutes. 

To write a sentence for any topic. 

To obtain a score of 50% on a final test for the course. 

Get a score of 50 or more on a 100 item final. 

Ccor.i better than at least half the class on the final teat in 
this course. 

Must be able to answer correctly at least 50% of the items on a 
100 question true- false test. 

25. A. Write the names of the Canadian provinces on an outline map. 

B. Write the ten provinces on an outline map provided in class. 

C. Write the names of at least 7 of the 10 Canadian provinces in a 
5 minute period. 

D. In five minutes write the names of ten provinces on a Canadian map. 

26. A. To underline verbs in sentences accurately. 

B. To locate and underline verbs in sentences correctly. 

C. To underline all verbs in 10 sentences in 15 minutes with 2 or 
fewer errors. 

D. To write all verbs from 10 sentences on a separate sheet of paper. 

27. A. By labeling a given outline map of waterways correctly within 

*5 hour . 

B. By being able to look at an outline map and locate waterwa 3 rs correctly, 
c. By placing waterways on an outline map accurately. 

D. By labeling without error ail the waterways on an outline map in 
30 minutes. 

28. A. Must compute accurately to 1 decimal place at least 20 of 30 

given division problems. 

B. Must work out long division problems in such a way as to demonstrate 
ability . 

C. Must finish accurately an assignment calling for solution of 
long division. 

D. Must be able to work 20 long division problems in 30 minutes. 

29. A. Must be able to keep time to a given record of music. 

B. Jftjst clap hands in 4/4 rhythm through ten bare of ”Ten Little Indians." 

C. ^st correctly clap in 4/4 rhythm, 4 counts in each measure, to 

a recording of "Ten Little Indians." 

D. Must be able to demonstrate the ability to keep time to a given record. 



23. A. 

B. 

C. 

D. 

24. A* 

B. 

C. 

D. 



V- 

ViV" 
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V. 



From each of the following groups of 'statements select those which describe 
a condition under which an objective Is to be measured. 



30. A. Must be able to Identify cones, cylinders, and prisms. 

B. Given a set of geometric chapes 

C. Within a period of 30 minutes with less than 3 errors 

D. Students In a 10th grade Geometry clasj 

I 31. A. Without the aid of references 

B. 33 correct out of a possible 50 
: C. 9th grade geography students 

D. Select the proper location of major rivers. 

32. A. Compute the area of a circle. 

B. Without the aid of a slide rule. 

I C. Following the proper formulas. 

D. 9th grade algebra students. 

f. 

33. A. Given a problem of the following class. 

B. Select the correct answer in 60% of the class. 

C. Be able to answer correctly 

D. The entire 12 grade calculus group 

34. A. In a period of less than 1 hour 

B. Without the aid of a reference map 

C. Find the location of a major continent 
D* Correctly In 40% of all cases 

I 35. A. By arranging parallel lines on a given map 

I B. The student will Identify and label 

I C. Three of the basic map projections 

I D. Will spell all three correctly In a period of 5 minutes 

; ( 

36. A. The student will solve an algebraic equation 

B. Given a linear equation with one unknown 

C. Within a period of 40 minutes 
And follow the correct procedures 
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Each of the following statements is a part of a behavioral objective. For 
each statement select the ansr^er which best describes i^at the statement 
refers to in the objective. 

37 . . .within a period of 20 minutes ... 

A. Audience 

B . Behavior 

C. Condition 

D. Degree 

38. ..the first year college geography class... 

A* Audience 

B . Behavior 

C. Condition 

D. Degree 

39. ..given a set of carpenters tools *. . 

A. Audience 

B . Behavior 

C. Condition 

D. Degree 

40. ..without the use of references... 

A. Audience 

B . Behavior 

C. Condition 

D. Degree 

41. ..all auto repair men in electrical circuiting will... 

A. Audience 

B. Behavior 

C. Condition 

D. Degree 

42 . . . locate and label . . . 

A. Audience 

B . Behavior 

C. Condition 

D. Degree 

43. ..with a slide projector and slides... 

A. Audience 

B. Behavior 

C. Condition 

D. Degree 

44. ..identify the areas containing salt, phosphorous and... 

A. Audience 

B. Behavior 

C. Condition 

D . Degree 












1 1 1 „ ii.^ivjipmpni 



mmmmm 



%' 









VII. 

The following behavioral objectivea are followed by a list of the four 
basic requirements necessary for a well stated objective. Select that 
requirement which you feel is least adequately met, or has been omitted 
altogether. 

45. The sixth grade social studies student, given a slate cutline map, 
will write the names of continrats on it. 

A. Audience 

B. Behavior 

C. Condition 

D. Degree 

46. Given roughly circular shapes with various arrows Indicating direction 
the student shall select without error in a 5 minute period al.l those 
whose arrows Indicate a cl£i>i!‘,fcwlse rotation. 

A. Audience 

B . Behavior 

C. Condition 

D. Degree 



47. The ninth grade social studies student will locate and name at least 
4 of the 5 climatic areas of Canada. 

A. Audience 

B. Behavior 

C. Condition 

D. Degree 

48. By the end of two months they should be able to type 20 words per 
minute for a period of five minutes with less than three errors. 

A. Audience 

B . Behavior 

C. Condition 

D. Degree 

49. On an outline map provided, ninth grade geography students will idcmtify 
and label the major rivers of the U.S. and Canada. 

A. Audience 

B. Behavior 

C . Condition 

D. Degree 
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Objective Malysls 
Criterion Test 



1. Which of these Is a benefit of pre»deslgn in contrast to 
exten^oraneous design? 

a) early planning Is diminished. 

b) restoration of the proper emphasis of teacher 
functions Is facilitated. 

c) student study tasks are more closely monitored. 



2. A programed Instructional booklet exemplifies: 

a) a superficial choice of terminal objectives. 

b) Intentional extemporaneous design of the conditions 
of learning. 

c) pre-design of the conditions of learning.- 

d) a wide range of enabling objectives. 



3. Pre-design of Instruction is a waste of time. 

a) agree 
by neutral 
cT disagree 



4. Since we do not know enou^ about the learning process, it is 
impossible to pre-design Instruction to make it more effective. 

a) agree 

b) neutral 

c) disagree 



5. The best use of an Instructor's time is to know his subject 
matter thorou^ly rather than to worry about how to teach it. 

a) agree 

b) nuetral 

“ c) disagree 

6. A student is cheated if he goes to a class when Instruction 



is not pre-designed. 


a) 


agree 


b) 


neutral 


c) 


disagree 
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Pre^deslgn Is a poor substitute for exteiaporaneoua design. 

a) agree 

b) neutral 

c) disagree 



Pre-design will do a lot to Improve Instruction. 

a) agree 

b) neutral 

c) disagree 



If X had my way, all Instruction would be pre-deslgned. 

a) agree 

b) neutral 

c) disagree 



In many subjects, extemporaneous design Is all rl^t. 

a) agree 

b) neutral 

c) disagree 



An objective analysis may produce a "good” hierarchy or a "not- 
so-good" hlerardiy. In the long run, a "good" hierarchy Is 
one that 

a) has many levels correctly positioned. 

b) Includes enabling objectives that are easily accom- 
plished by learners. 

c) help the Instructor In deteinnlnlng what to teach, and 
In what sequence to teach It. 






/ 

0 

/» 



12. Which of these terms does describe '‘enabling objectives”? 

a) Basic factual and conceptual knowledge. 

b) Component behavior or skills. 

c) Lower order competencies. 

d) Format of the instruction. 



13. After identifying the enabling objectives the next step is to: 

a) Identify the type of learning Involved in each 
objective. 

b) Develop an instructional environment which will 
transform learners into graduates who can perform 
at the specified level. 

c) Design instructional sequences that reflect the 
basic learning principles. 



14. After establishing the terminal objective(s) the designer 
should ask the following question. 

a) What level on the hierarchy of learning is this 
objective? 

b) What media will best facilitate reaching this goal 
without losing the individualized approach? 

c) What would the student have to know to perform this 
objective successfully? 

d) What in the world do X do now? 



15. Below is a set of objectives. Two of these are enabling 
objectives. One is a terminal objective. 

1) Kep resent forces and their directions as parts 
of triangles. 

2) Name parte of a triangle. 

3) Name horizontal and vertical components of forces 
as vectors . 

Which is a terminal objective? 

a) 1 

b) 2 

c) 3 
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16. Betor it» a set of objectives. One is a terminal objective 
and the others arc enabling objectives. 

1) The student will make a left turn with a stan- 
dard shift car onto a highway from a stop. 

2) The student will discriminate the clutch from 
the gas pedal and the brake. 

3) The student will disengage the clutch. 

4) The student will signal for a left turn. 

Which of the following is a terminal objective for this 
set? 



a) 1 

b) 2 

c) 3 

d) 4 

17. In the following objective analysis, choose the correct 
ordering of enabling objectives. 




1. Distinguish between equal 
and unequal expressions. 

2. Perform operations of 
multiplication and division 

3. Recognize false math 
sentences 

4. Recognize the math sentences 

5. Recognize true math 
sentences 



Ila - 
Ilb - 
III - 
IVa - 
IVb - 



a . b . 

1 2 

4 1 

2 4 

5 3 

3 5 



c. d. 

5 3 

3 5 

2 4 

4 1 

1 2 




i ■ 

18. Bead the following list of objectives for a math problem. 

1) Read 1 and 2 place numerals. 

2) Multiply a two place whole number by a one place 
whole number. 

3) Multiply a one place whole number by another one 
place ^^ole number. 

4) Add a group of ten(s) to a two place whole nuiober 
(carrying) • 



Which diagram Is closest to revealing the hierarchy? (The numbers 
In the squares refer to the numbered objectives above.) 
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Prototype Development 
Criterion Test 



For test Items 1-8, circle the answer you consider best* 
then push the same letter button on your response station. 

1. The 8tap,es defined in the instructional system develop- 
ment section are: 

A. Identify behavioral objectives; determine enabling 
objectives, produce the instructional product; 
analyze tryout results. 

B. Produce instructional product; conduct tryout of 
product; analyze tryout results; modify product; 
recycle . 

C. Determine enabling objectives; produce instructional 
product; construct performance measures; conduct 
tryout; analyze results; recycle. 

D. Produce instructional product; construct performance 
measures; conduct try-out; analyze results; recycle. 

2. The process of systematically following the instructional 
’’blue prints’' or specif icatidns is to: 

A. Produce the instructional product 

B. Identify types of learning 

C. Identify behavioral objectives 

D. Conduct try out of the product 

3. To translate the analysis of try-out results into changes in 
the Instructional materials is to; 

A. Determine enabling objectives 

B. Produce the Instructional product 

C. Recycle the development process 

D. Modify the product 

A. Obtaining data from observations and other evaluations of 
the system such that weaknesses can be Identified is to: 

A. Construct performance measures 

B. Determine enabling objectives 

C. Conduct try-out of the pr^niy; 

D. Analyze try-out results , 
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5. The stage of instructional system development vhose purpose 
is to determine whether the instructional system achieves 
its objective is the: 

A. Analysis of try-out results 

B. Identification of events that provide conditions of 
learning 

C, Detemlnatlcn of enabling objectives 

D, Production of the instructional product 

6. Recycling is concerned with 

A. Determining enabling objectives for all behavioral 
objectives 

B. Administering and analyzing performance measures 
Modifying all phases of th© Instructional product 

I). Reduplicating all instructional system development 
stages 

7. The team of specialists considered essential to produce the 
instructional product Includes a: 

At Content specialist, media specialist, behavioral 
scientist 

B. Production specialist, measurement specialist, field 
monitor 

C. Subject matter specialist, field monitor, liaison 

D. Production specialist, try out specialist, measurement 
specialist 

8. The continuum of experience model is most useful in 

A, Constructing performance measures 

B, Interpreting results of the trv-out 

C, Deciding what media to use 

D, Deciding how to conduct the try-out 

For test items 9-16, circle whether the statement is true 
or false and push the sam.e letter button on your response station. 

9. Try-out of the instructional product occurs after all develop- 
ment has been completed. 

A. True 

B. False 












10. Try-out of the instructional product should only be conducted 
with learners of appropriate prade and ability level. 

A. True 

B. False 

11. Analysis of try-out results cannot put more confidence In 
test results than In consents from try-out subjects. 

A. True 

B . False 

12. Pi^h error rates on a criterion test indicate the Instruc- 
lonal system has portions that are relatively Ineffective. 

A. True 

B. False 

13. Try out following Instructional system modification Is 
crucial . 

A. True 

B. False 

14. If it is possible, the same subject should be used to try- 
out all revisions of a particular segment of the new In- 
structional system. 

A. True 

B. False 

15. Product development requires translating instructional 
specifications into written statements and/or media forms. 

A. True 

B. False 

16. Incidental details having little to do xjlth the basic 
content of the new Instructional system often Interfer with 
effective learning during the try-out. 

A. True 

B. False 
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For test items 17 - 23, circle the answer you consider best; 
when all persons have completed these items you will be in- 
structed to push the appropriate button on your response station. 

17. Analysis of try-outs is used to: 

A. Plan modifications to the system 

B. Determine weak teaching strategies 
0. Determine unrealistic specifications 

D. All of these 

E. Mone of these 

18. To measure the first try-out of a segment of a net»J instruc- 
tional system 

A. reauires a validated test 

B. Demands small groups 

C. Involves only learners 

D. All of these 

E. None of these 

19. Modifications to the new instructional system follot^ing 
analysis are planned with: 

A. The same team of specialists used in production 

B. A new team of specialists 

C. The content specialist only 

D. All of these 

E. None of these 

20. The process of producing the instructional product ii- 
volves 

A. Systematically following the instructional specifi- 

cations 

B. Translating specifications into prototype 

C. liaison between. a team of specialists 

D. All of these 

E. None of these 



mm 









21. To modify the instructional materials following try-out 
requires 

A. Translating the analysis results into changes 
in the product 

B. Planning with the development specialist team 

C. Revising enabling objectives 

D. Ml of these 

E. None of these 

22. In teaching learners to discriminate between sounds 
produced from an oboe and an English horn, according to 
the continuum of experience model, use 

A. Direct experience 

B. Objective codification (audio) 

C. Subjective codification (audio) 

D. All of these 

E. None oi these 

23. Results of an analysis of the total instructional system 
indicates that learners achieved all the stated enabling 
objectives but did not achieve the terminal objective. 
Which, interpretation would be appropriate 

A. Enabling objectives were Irrelevant to the terminal 
objective 

B. Enabling objective stated required learning steps 
that were too big. 

C. Enabling objectives were too easy 

D. Ml of these 

E. None of these 
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During the first session in the Institute on Specifying 
Behavioral Objectives you x^^rote two objectives in the 
area of your discipline. For this part of the test, 
copy one of those objectives in the space below and do the 
following: i^lrst, identify one enabling objective 

necessary to attain the terminal behavior you stated. 

Then: 

A. Describe how you would proceed to produce the in- 
structional segment for that enabling objective. 

B. Detail how you ’Tould plan to try-out the segment pro- 
duced. In your discussion give consideration to all 
pertinent aspects of production and try-out covered 
in this section. 
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25. The section that follows contains a series of frames taken 

from a self -instructional program designed to teach 6th grade 
students concepts and principles of latitude end longitude. 
Belotf the frames is a summary of the error rates for each 
frame, resulting from a tryout of the materials with low, 
average and high ability students drawn from the 5th, 6th 
and 7th grades. After studying these materials, 

A. Judge which frames were bad for vrhich students. 

(Consider acceptable frames as those not exceeding 
an average error rate of 14.) 

B. Write a brief statement (approximately 100 words), 
based on the evidence available, indicating what 
you think caused the frames to be faulty. 

C. Briefly describe the steps you would take next to 
modify the faulty frames. 
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Figure 10 



8-30 Look at Figure 10. The longitude of Point A 
la written (?) 

8-31 The longitude of Point B on Figure 10, is 
wrltten^^ (?) 

8-32 The longitude of Point C on Figure 10 Is 
writte n (?) 

8-33 Longitude is (?) - (?) distance or 
location. 



160°E 

120°W 

120®E 



east-west 



8-34 Longitude lines are numbered from 0 at the 

(?) meridian to (?) ^ prims; 180° 
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8-35 On Figure 11 we have an imaginary map with 
with imaginary cities. See if you can read 
their longitudes. Use Figure 11 for the rest 



of this set. The longitude of Conlin is 

^ 3% 

8-36 The longitude of Elten is (?) ^ (?) * (?) . 1°30*W 

8-37 The longitude of Dagman is (?) ° (?) * (?) 0°45*W 

8-38 The longitude of Frantin is (?) . 1®15*E 

8-39 The longitude of Baatle is . 1®15*W 



Error Rates Per Frame by Grade and Ability Level 



Frame 

Number 
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Low 

N-18 


5 

Ave. 

N-16 


5 

High 

N-18 


6 

Low 

N«9 


6 

Ave. 
_ t!?19 


6 

High 

N-20 


7 

Low 

N«12 
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Ave. 

N-14 
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High 

N-5 
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A. Identify those frames judged bad and for Trhich students 



Bad Frames 



T'%ich Ftudents 



1 



i 

I 

I 



B. What caused frames to be faulty? 



C. ^IhRt next steps would you tske to modify faulty frames? 
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TEST OVER TOPIC V: MEASUREMENT 



/ 



Part I . Foundations 

Items appearing in Part I of the test are designed to assess knowledge that is basic 
to tlie use of measurement in instructional systems development, evaluation, and research. 

In responding to the first set of items place a T in front of the items which are true 
and an in front of those which are false. 

1, Demonstrating the effectiveness of an instructional system in one school setting 

nearly always means that it will be effective in another. 

2. The aim of instructional evaluation is to determine whether a particular 
instructional system brings about the educational outcomes (behavioral objectives) 
that arc intended to derive from it. 

3. Because of the ’’inexactness" of measurement in education and the behavioral 
sciences two or more measures should be used to assess any characteristic or 

4. The essential difference between measurement in the physical sciences and the 

behavioral sciences lies in the fact that physical sciences rely upon direct 
measurement, 1 . c. , measurement of observable properties, while the behavioral 
sciences rely upon in direct measurement, i.e., measurement of "indicants" of 
inferred properties. 

5. The control of "random" error in measurement is one of the moat bothersome 

and difficult problems that educators face in carrying out instructional 
research and evaluation. 
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T!ie second set of Itmes in Part I of the test Involve written reaponeee. In writing 
these responses keep them brief. 

1. A distinction has been made in the chapter between instructional research and instructional 
evaluation. In what way do the problems of measurement differ for the two? 



Three distinctions have been made in the present chapter that are basic to essentially all 
of the thinking that has been outlined in it: (a) the distinction between low inference 

and high inference measures, (b) the distinction between measures which require only the 
recording of the data that derive from them and those that require coding or scoring in 
classification prior to recording, and (c) the distinction between evidence needed in 
support of the trustworthiness of a measure and evidence needed in support of the accuracy 
with which a measure is applied. What is the advantage or power that derived from each of 
these distinctions? 

Low vs. high influence measures: 



Measures which require only recording of the data that derive from them v£. those that 
require coding and recording: 



Evidence of the trustworthiness of a measure evidence of the accuracy with which a 
measure is applied: 
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The position has been taken in the Chapter that any measure, regardless of its level of 
instrumentation and inference, must have five kinds of evidence marshalled In Its support 
before it can be accepted as trustworthy. These classes of evidence have been labeled 
respectively, relevance, representativeness, fidelity, accuracy, anJ consistency. For 
each of these characteristics (a) define or describe it in such a way as to make clear 
the dimension of measurement it taps, and (b ) Indicate how such evidence is obtained. 

Relevance: i 



Representativeness : 



Fidelity : 



i 

I 



. / 

f 

i 

1 



I 

Accuracy: 



* 

i 



Consistency : 
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Part II. Applications 

Items appearing in Part II of the test are designed to assess the ability to apply 
basic knowledge about measurement to the solution of tasks demanding of that knowledge. 
In responding to the first item identify the elements (properties, traits,, character- 
istics, outcomes) that can be measured only indirectly , i.e., that are hypothetical 
constructs . Do this by placing a check in front of these elements. 



Ability to solve story problems in matliematics 



Knowledge of ancient history 



Personality 

Intelligence 

Anxiety 

Time required to run the hundred yard dash 
Considerateness 



Orientation to the use of questions 
Citizenship 



5 
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In responding to the second item In Part II of the test Indicate the level(s) of 
measurement (column 2) that are appropriate and feasible within a school setting for the 
measurement of the cognitive outcomes listed in column 1. Do this by placing the number 
descriptive of the level (s) of measurement in front of the cognitive outcome that is to be 
measured. Where more than one level is appropriate rank them according to the power of the 
data that wou l d come from each , listing the moat powerful first. 



Rules governing ba sketball 



1 . Knowledge level 



History of the Eskimo 2. Situational response 

level 1: Described 

criterion behavior 

Sensitivity to the feelings of others 



Competence in math 



Competence in marbles 



Use of good table manners 



3. Situational response 
level 2: Related 

criterion behavior 



4. Situational response 
level 3: Isomorphic 

criterion behavior 



Reading skill 



Writing skill 



Self confidence 



Tolerance of individual differences 
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Part III. Instrument Development 

Items appearing in Part III of the test are designed to Assess knowledge Is 

required in order to develop instruments for use in educational measurement. In responding 
to the first item, match the various attributes of a measure that are listed in column 1 

with the phrase in column 2 that most appropriately describes the kind of evidence required 
in support of that attribute. 



Relevance 



Representativeness 



Fidelity 



Consistency 



Construct validity 



Accuracy 



1. Theoretical clarity of items 

2. Conceptual appropriateness of Items 

3. An individual's scores on a test taken 
twice with two weeks Intervening 

4. An Individual's scores on alternate forms 
of a test 

5. The correlation between an individual's 
performance on two measures that are known 
to be related 

6. The extent to which an item differentiates 
between those scoring high and low on a tesi 



7. The extent to which an item calls for 

behavior that is isomorphic to the property 
that is being measured 
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In reaponding to the second Item in this part of the test indicate the clas8(es) of 



roeasurement (column 2) that are appropriate and fe asible within a school setting for the 
rreasurefnent of the cognitive outcomes listed in column 1. Do this by placing the number 
descriptive of the class(es) of measurement in front of th-* cognitive outcome to be measured. 



l/hore more than one class is appropriate r ank them according to the power of the data that 
wo uld come from each , listing the most powerful first. 



Rules governing basketball 



History of the Eskimo 



Sensitivity to the feelings of others 



Competence in math 



Competence in marbles 



Use of good table manners 



Reading skill 
Writing skill 
Self confidence 



Tolerance of indlvidiisl differences 
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1. Interview 



2. Systematic observation 



3. Standardized objective measures 
(Paper and Pencil) 



4. Standard!* .d projective measures 
(Paper and Pencil) 



5. Teacher-made tests (Paper and Pencil) 



6. Unobtrusively obtained products 
and/or records 



7. Unobtrusive observation 
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CRITERION TEST; EXPERIMENTAL DESIGN 



Directions 2 Read the proposal extract and answer the questions 
that follow. 



111. ObJ ectivas 

The primary objective of the proposed research is to determine 
whether or not student attitudes towards teaching objectives in- 
cluded in teacher education programs are effectively altered after 
they have had experience in simulated settings > The experimentation 
will be designed to answer questions which are specific to the 
particular experiences included in the materiols. The simulation 
problems are, in this case, limited to situations of a personal- 
social nature. 

Specific questions to be answered are the following: 

I. Will the experiences provided through individualized 
Instruction in the simulation facility (as described by Kersh, 1963), 
product! changes in attitude in a positive direction toward the 
subject -matter of educational psychology? 

II. Will changes in attitude become evident after the learner 
has had experience with one progroiii of 20 problem sequences? 

'3. Will changes in student attitude maintain in strength 
following the. termination of the learner's experience In the 
simulated setting? 



iV. Procedure 

A. General Method and Research Strategy 

1. Sa mple Fl an and Pr ocedure. The experimental subjects will ? 

be selected from the students at Oregon College of Education ! 

majoring in elementary education. They will complete their in- 
struction in the classroom simulator within one academic year of 

their practice- teaching experience or internship. The subjects I 

wil.l be screened on the basis of previous educational backgro.und 
and experience to insure homogeneity in this regard. Also, they 
will be screened on the basis of their performance on the pretest 

of attitude toward specific teaching objectives selected from those 1 

Included in education courses. In so far as possible, only those ^ 

students whose performance on the pretest indicates that they are 1 

neutral or negative toward the specific teaching objectives will be 
Included. Subsequently, the subjects will be assigned to either 
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the experimental or the control group rant|omly on the baaia of 
acholaatlc aptitude, aex, and other aelecf^ed varlablea which other** 
wiae might bias the findings. 

2. Treatments . The classroom simulation technique under con~ 
sideratloh attempts to create for the etudent teacher all of the 
relevant features of a single classroom situation. 

In brief, the instructional procedure which has been developed 
for experimentation with the simulation materials is as follows: 

First the student teacher (T) is oriented to tho simulation facility 
and to the procedure, then he is given a performance test in the 
simulated classroom using one of the three programs . The orlenta** 
tlon and pretesting procedure takefli approximately one and one>half 
hours per subject. Next, T is given the actual instruction in the 
simulation facility. The filmed problem sequences of actual class** 
room situations are presented and T is requested to enact his res** 
ponse to each. Depending upon the reaction of T, Che expetiaencor 
(£) selectee and projects one of two or three alternative feedback 
sequences . 

3. Controls . The control group will be treated in every 
respect the same as the experiment&l group except the individuals 
in the control group will not experience Instruction in the sia** 
ulated setting. Differences in attitude between the experimental 
and control groups may be attributed only to experience in the 
simulation facility. 

B. Data Types to be Gathered and Methods to be Used. 

i 

As is Indicated above, the experimental group will undergo | 

experience in the simulation facility which may be conveniently 
interrupted at three, points: (1) at the end of the pretest period 

which includes the first 20 problem sequences; (2) after the in** 
structional period, and, (3) after the post*test which terminates 

the experience in the classroom simulation facility. Criterion j 

tests will be administered to individuals in the experimental group 

before their experience in the simulation facility begins, and after 

each of the instructional phases Indicated above. Approximately 

six weeks after termination of the simulation experience,, the 

criterion test will be administered once again*. Individuals in | 

the control group, which will not have experience in the simulation 
facility, will be tested on a time schedule approximating that of 1 

individuals in the experimental group. 
















QUESTIONS: 

1. Whet is ths sxpsrlmsntAl unit? 

2. Whst is/srs ths sxpsrimsntsl trsstmsnt (s)7 

3. Whst is ths Bsthod of sssigiu&snt? 

4. Disgrsm ths dsslgn using R'i» X's» and O's* 

5 . List sourcss of invalidity that ars acsountsd for. 

6 . List sourcss of invalidity that ars not accountsd for. 



7. Rsdssign ths proposal, givs both nsw diagram and brisf 
sxplanation. 
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Data Analysis I 
Criterion Examination 



1. Assume that the numbers 6 , 13, 24 were taken from a ratio 
scale. Which of the following sets of numbers would be 
appropriate substitution? 



a. 


13, 20, 31 


b. 


24, 52, 96 


c. 


11, 25, 47 


d. 


1, 6, 14 


If the numbers 4, 


17, 43 were taken from an ordinal scale. 


which of the following sets would be an appropriate substi- 
tution? 


a. 


1, 2, 3 


b. 


4, 16, 12 


c . 


13, 10, 26 


d. 


none are appropriate 


If the nxunbers 6, 


21, 24 represent an Interval scale which 


set of nuodDers would be an appropriate substitution? 


a. 


11, 40, 45 


b. 


12, 42, 48 


c • 


18, 63, 72 


d. 


21, 35, 39 



For each of the following kinds of measures Indicate whether the 
scale is nominal, ordinal, Interval, and ratio 

4. Human age in years. . . 

5. IQ ■ 

6. Numbers on pages In a book 

7. Notes on the musical scale 

3. Temperature on a centigrade scal e 

9. Elapsed time In seconds 

10. Calendar year& . . - 




11* An Investigator wished to determine whether physical coordi- 
nation training was effective In improving reading 
effectiveness. He selected a sample of “below grade place- 
ment" readers and gave them 20 hours of training in lateral 
coordination. A second sample was chosen from the same pool 
of low readers and given no training. For analysis purposes 
he should consider the samples 

a. Independent 

b. related 

12. A second investigator studying the same problem selected only 
one group of low readers and gave them training. He then 
compared pretraining and post- training reading performance. 
His samples should be considered 



13. A third investigator thought that differences In training 
effectiveness might be expected for boys and girls. He 
designed his study to have training and no training groups of 
boys and girls. How many samples would be required for 
analysis purpoRes? 



14. The null hypothesis which states that "the differences 

between means and variances of samples are no greater than 
differences due to the vagaries of random sampling from a 
single, normally distributed, imfinlte population," is best 
suited for 



a. causal-comparative studies 

b. experimental sbwdies 

c. exploratory studies 

d. studies of physical characteristics only. 

15. Causal-comparative studies differ from experimental studies 

in that 

a. Causal-comparative studies look fo\* differences while 
experimental studies only look for change. 

b. Causal-comparative studies attend tc differences In 
means and variances while experlmenti;^! studies attend 
only to differences In means. 

c. Causal-comparative studies seek explanations on the 
basis of some characteristic of subjects, while 
experimental studies seek to determine affects of an 
additional influence • 

d. The two are synonymous. 



16. All too often investigators state their null hypothesis 
merely as "the samples did not differ." This leads to 
confusion becausfs 

a. The samples are not adequately described. 

b. The dependent variable is undefined. 

c. The analysis technique is unstated. 

d. The basic assumptions of sampling are omitted. 

For each of the following research problems indicate which null 
hypothesis (I, II, or III) is most appropriate. 

17. An investigator wished to determine if right-handed or left- 
handed baseball pitchers revealed greater eccentricities as 
measured by Karl Olliver Obvious Key Induction Examination. 
He chose random samples of 50 right-handed and left-handed 
pitchers from rosters of the teams in the American, 

National, and Pacific Coast leagues. 



18. After select! ’ h- three random samples of sixth grade students, 
an Invest igi gave one group Intensive training in problem 

solving, a second group a similar amount of training in logic, 
and no tro.ining to a third group. then tested the creative 
??,bilitiGs of tViG thr^&c groups. 



19. An Investigator wished to determine whether school admini- 
strators were more effective after completing his course. 

He watched his class members with other administrators In 
his state on the basis of sex, age, years of administrative 
experience, and number of hours of graduate credit. 



20. An Investigator wished to verify his observation that boys 
were more quantitatively inclined than girls. He selected 
random samples of 10th grade boys and girls enrolled in 
his high school. 
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Data Analysis XI 
Criterion Examination 



For each of the following research situation^ describe the 
appropriate analysis technique. 

1. The Investigator wished to determine whether or not achieve- 
ment in arithmetic, as measured by a standardized test 

reporting scores In grade placement equivalents was related 
to nonverbal I.Q, 



2. An Investigator asked the question "is college attendance 
predicted by high school grade point average, size of 
graduating class, and SAT verbal scores?" 



3 . 



A group of judges observed teaching behavior exhibited b*^ 25 
student teachers and ranked them according to their effec- 
tiveness. The Investigators wished to determine if these 

rankings were related to average grades In professional 
courses received by the subjects. 



4. An investigator determined the validity of his test by 

showing that persons who scored higher on his test completed 
their degree programs while those who scored low^^r did not 
complete their degrees. This relationship could beat be 
described by: \ 



5. To evaluate the effectiveness of a set of instructional 
materials, the researcher adrainistcred pre- and post- 
training tests (25-iten, multiple-choice’ test) to a 
group of college freshman English students. 



6. A college student personnel officer wished to know whether 
purchase of activities tickets differed among students 
living in dormitories, living in off-campus apartments 
or commuting from home. 



it 

I 
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7 . 



Reaction times were recorded for three types of students, at 
four different times of the day, under two teinperature 
conditions for five consecutive days. 



8* A college political science professor was Interested in 

determining whether men or women tended to participate more 
frequently In anti-Viet Nam demonstrations held on campuses. 

11 students enrolled In political science courses were 
polled. 



^ i 



9. A biology instructor teaching two sections of bacteriology 
required weekly lab attendance for members of one class and 
compared their performance on his final exam with performance 
of the second class which experienced no lab sessions. 
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m 



10 . 



i 



A sales manager was Interested in determining which of 
several characteristics (annual IncoTue, age, size of 
family, etc.) were most accurate In differentiating 
Cadillac buyers from Corvair buyers. 



I 



11. A dean of women postulated that women enrolled in the College 
of Education were more attractive than women enrolled in the 
College of Arts and Sciences. Over a five year period she 
recorded the college affiliation of all women who were 
finalists in the 10 separate beauty contests held each year. 



12. A curious physiological psychology major wished to determine 
whether scholastic aptitude (SAT verbal score) was related 

to sex (male and female) . ^ 
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X3. A college golf coach wished to determine whether there was a 
relationship between the order of finish In his conference 
tournament and season average scores of his team members. 



14, Ten pairs of identical twins were identified by a psycholo- 
gist. At random one twin from each pair was assigned to 
attend nursery school for one term. At the end of the term 
the psychologist obtained measures of social adaptivity for 
all 20 twins. Scores on the social adaptivity scale were 
such that they could be ranked In order of absolute 
magnitude. 



15. An Investigator wishes to determine whether the 20 Items on 
his test differ In difficulty. He administers the test to 
75 subjects and records pass-fall information for each itme 
for each subject. 
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16. An Investigator wished to determine whether there was any 
association between social class status and choice of high 
school curriculum. In a large high school he recorded the 
number of students in each of five social classes which 
chose each of three curricula. i 
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APPENDIX D 



Written Evaluation Form 



APPENDIX D 



NATIONAL RESEARCH TRAINING INSTITUTE 
EVALUATION QUESTIONNAIRE 



Your constructive comments can directly benefit future participants 
in this institute, by helping the staff determine which of the elements 
in our instructional program should be retained in their present form, 
which should be modified, and which should be eliminated. Thank you f<»r 
your assistance and cooperation. 



(1) What instruction did you find aost relevant to your professional 
needs? 



(2) In what ways will the things you described in (1) influence the 
performance of your job in the coming year? 



(3) What could be added or changed to make the Instruction more use** 
ful and effective? 



(4) What do you feel should be eliminated or de'*emphasized? 
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(5) Please write at least one positive and one negative comment on 
each of the following: 

(a) Reading materials 

(b) Small group activities 

(c) Flexibility of the program 

(d) Interaction with staff and other participants 



(6) On the first day of the institute, you wrote a paragraph on 
some point of disagreement with the general orientation 
chapter. How if at all, would you modify now what you wrote 

then? 



(7) There must be something we've overlooked in our evaluation. 
Please answer (a) or (b) below: 

(a) You must be kidding! 1 have writer's cramp already. 

Check here. 

CZ] 

(b) What I really wanted to tell you, but you never asked, is 



